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LONG-TERM  GOALS 

The  APL  atmospheric  sciences  group  is  working  to  improve  forecaster  performance  at  Navy 
operational  weather  forecast  detachments  afloat  and  ashore.  This  work  encompasses  broad  research 
and  technology  development  in  areas  of  visualization,  human  factors,  human-machine  interaction,  and 
model  and  forecast  verification  with  an  emphasis  on  mesoscale  ensembles  and  visualization  of 
uncertainty.  The  verification  effort’s  long-term  goal  is  to  develop  an  automated,  objective  verification 
technique  for  assessment  of  very  high-resolution  mesoscale  predictions  which  accurately  accounts  for 
spatially  or  temporally  misplaced  features,  false  alarms  and  misses  (Brown  2002). 

OBJECTIVES 

The  overall  objective  of  this  effort  is  to  develop  a  highly  automated,  rapid,  object-oriented,  mesoscale 
numerical  weather  prediction  (NWP)  verification  tool  for  use  by  forecasters  and  model  developers. 

The  verification  technique  should  consider  distortion  errors  (phase/timing,  rotation,  and  stretching)  as 
well  as  the  normal  amplitude  errors.  It  is  intended  to  test  the  verification  tool  on  the  University  of 
Washington  Short  Range  Ensemble  Forecast  System  (SREF)  and  with  a  version  of  the  Navy  CO  AMPS 
model  implemented  at  APL. 

APPROACH 

Recently,  a  number  of  techniques  from  spatial  statistics  have  shown  promise  in  verification  problems 
(Marzban  and  Sandgathe  2005;  Harris  and  Foufoula-Georgiou  2005).  Among  them  there  exists  one 
technique  which  not  only  allows  for  a  novel  verification  method,  but  also  addresses  a  central  issue  in 
our  mesoscale  verification  technique  (MVT)  -  the  “box  size”  (or  characteristic  length  scale  as 
discussed  in  Hoffman  et  al  1995  and  Nehrkom  et  al  2003).  The  tool  is  the  variogram  and  our  proposal 
is  to  use  the  variogram  to  address  the  length-scale  issue,  both  for  verification  and  to  address  the 
‘feature’  background  error  covariance. 
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A  variogram  is  a  diagram  that  displays  the  covariance  (or  correlation)  between  pair  of  points  as  a 
function  of  the  distance  (and  direction)  between  them.  In  Spatial  Statistics  it  is  employed  to  quantify 
the  covariance  structure  of  a  spatial  field.  The  variogram  can  be  employed  for  the  following  three 
purposes: 

1)  for  verification  purposes, 

Like  a  power  spectrum,  a  variogram  quantifies  the  various  scales  present  in  a  field.  From  an  image- 
processing  point  of  view,  a  variogram  assesses  the  texture  of  an  image.  When  applied  to  a  forecast 
field  and  an  observation/analysis  field,  it  provides  a  means  of  comparing  the  two  fields  in  terms  of 
their  texture,  or  the  distribution  of  energy  among  the  various  scales.  Harris  and  Foufoula-Georgiou 
(2005)  were  the  first  to  propose  variograms  as  a  verification  tool. 

Various  features  of  a  variogram  can  be  distilled  (quantified)  in  order  to  allow  automation  of  the 
verification  procedure.  Three  such  features  which  also  have  physical  significance  are  the  nugget,  the 
sill,  and  the  range.  The  nugget  refers  to  the  y-intercept  of  the  variogram  (or  the  model  fit  to  it);  the  sill 
is  the  value  along  the  y-axis  at  which  the  curve  levels  off,  and  the  range  is  the  corresponding  value 
along  the  x-axis.  The  slope  of  the  rising  portion  of  the  variogram  can  also  be  considered  as  another 
scalar  verification  measure. 

2)  to  provide  an  objective  assessment  of  the  length  scale  to  be  used  in  MVT,  which  in  turn,  is 
employed  for  verification. 

Through  their  implicit  treatment  of  scales,  variograms  also  provide  an  objective  estimate  of  the 
characteristic  length  scale  in  a  field.  This  length  scale,  in  turn,  is  related  to  the  “box  size”,  a  quantity  in 
MVT  that  is  often  determined  in  an  ad  hoc  or  subjective  manner  (Hoffman  et  al  1995  and  Nehrkom  et 
al  2003). 

3)  to  provide  an  alternative  estimate  of  the  background  error  covariance. 

The  issue  of  characteristic  length  enters  the  estimation  of  background  error  covariance  matrix  (“B”). 

As  such,  variograms  can  also  yield  estimates  of  B,  but  in  a  way  that  is  more  conducive  to  a  verification 
scheme  based  on  variograms.  Sen  (1997)  has  shown  the  utility  of  this  specific  use  of  variograms. 

WORK  COMPLETED 

New  effort. 

RESULTS 
New  effort. 

IMPACT/APPLICATIONS 

Verification  systems  need  to  be  highly  automated  in  order  to  rapidly  assess  large  samples  of  cases; 
however,  they  must  also  be  able  to  correctly  evaluate  the  high  frequency,  high  amplitude  signals  of 
mesoscale  features.  Unfortunately,  traditional  methods  of  verification  have  been  shown  not  to  work  for 
mesoscale  numerical  weather  prediction.  Simple  automated  techniques  incorrectly  assess  slight  phase 
or  displacement  errors,  causing  smoothed,  or  ensemble  mean  forecasts  to  appear  to  perform  better  than 
a  more  detailed  deterministic  forecast,  yet  they  contain  less  forecast  content.  Case  studies,  while  more 
revealing,  are  too  time  consuming  to  assess  the  large  number  of  cases  required  for  subtle  model  biases 


or  differences  arising  from  small  changes  in  model  algorithms.  Mesoscale  NWP  forecast  verification  is 
a  critical  issue  for  US  Navy  operations.  More  NWP  model  outputs  are  becoming  available  from 
various  sources  and  it  is  difficult  for  management  and  operational  forecasters  to  choose  the  appropriate 
system  for  each  forecast  situation.  The  forecast  verification  tool  will  enable  more  accurate  and 
meaningful  evaluation  of  mesoscale  numerical  weather  prediction  systems,  especially  mesoscale 
ensemble  systems,  where  large  volumes  of  data  are  required  for  accurate  assessment  and  where  small 
prediction  distortions  or  displacements  cause  significant  misinterpretation  of  verification  results.  The 
tool  is  intended  for  use  both  by  model  developers  and  by  forecasters  for  quick  and  more  accurate 
model  assessment.  The  forecaster  tool  will  be  implemented  as  an  easy  to  use  web  tool. 

RELATED  PROJECTS 

The  University  of  Washington  Multidisciplinary  University  Research  Initiative  (MURI)  on  Integration 
and  Visualization  of  Multi- Source  Information  for  Mesoscale  Meteorology:  Statistical  and  Cognitive 
Approaches  to  Visualizing  Uncertainty.  This  project  incorporates  a  number  of  verification  techniques 
into  a  forecaster  visualization  tool  and  a  prototype  version  of  our  mesoscale  verification  tool  has  been 
implemented  here.  The  Environmental  Visualization  (EVIS)  effort  of  the  Knowledge  Superiority  FNC 
funded  by  ONR  has  reviewed  the  MVT  and  is  planning  to  test  it  as  a  potential  user  application. 
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Figure  1.  Automated  Mesoscale  Verification  Tool  GUI 
[Automated  Mesoscale  Verification  Tool  (AMVT)  Graphical  User  Interface  displaying  model  and 
parameter  selection  for  web-based  submission  of  large  and/or  complex  verification  of  multiple 

models  or  multi-member  ensembles.] 


A  cluster  analysis  technique  (Marzban  and  Sandgathe,  2006a)  using  agglomerative  hierarchical  cluster 
analysis  which  incorporates  both  spatial  relations  and  parameter  value  (e.g.  precip  amount)  has  been 
tested  for  verification  of  discontinuous  fields  such  as  precipitation.  This  technique  was  very  well 
received  at  the  Montreal  Verification  Workshop  (Marzban,  2004)  and  has  been  funded  by  NSF  for 
rapid  transition  into  the  DoD/NOAA/NCAR  mesoscale  WRF  Developmental  Testbed  Center  (DTC).  A 
follow  on  paper  (Marzban  and  Sandgathe,  2006b)  has  expanded  this  technique  to  allow  joint  clustering 
of  forecast  and  observation  fields  to  provide  a  better  assessment  of  false  alarms  (forecasted  but  not 
observed)  and  misses  (observed  but  not  forecast)  precipitation  areas. 
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Figure  2.  Example  of  precipitation  field  decomposed  using  agglomerative  cluster  analysis. 
[The  field  on  the  left  is  a  plot  of precipitation  intensity  for  the  NE  Pacific  and  NW  America.  The 
field  on  the  right  is  the  same  field  indicating  clustering  of precipitation  based  on 

locationand  intensity.] 


REFERENCES 

•  Brown,  B.,  2002:  Development  of  an  Object-based  Diagnostic  Approach  for  QPF  Verification. 
USWRP  Science  Symposium,  April  2002. 

•  Chan,  E.,  1993:  Review  of  Block  Matching  Based  Motion  Estimation  Algorithms  for  Video 
Compression.  CCECE/CCGEI. 

•  Daley,  R.,  1991 :  Atmospheric  Data  Analysis.  Cambridge  University  Press,  New  York,  USA. 

•  Harris,  D.,  E.  Foufoula-Georgiou,  K.K.  Droegemeier,  J.J.  Levit  2001:  Multi-Scale  Statistical 
Properties  of  a  High-Resolution  Precipitation  Forecast.  J.  Hydrometeorology.  2,  406-418. 

•  Hoffman,  R.  N.,  Z.  Liu,  J.-F.  Louis,  and  C.  Grassotti,  1995:  Distortion  representation  of 
forecast  errors.  Mon.  Wea.  Rev.,  123,  2758-2770. 


•  Marzban,  C.,  2004:  Statistical  cluster  analysis  for  verification  of  spatial  fields  International 
Verification  Methods  Workshop,  September  15-17,  2004,  Montreal,  Canada. 

•  Reilly,  C.,  P.  Price,  A.  Gelman  and  S.  A.  Sandgathe:  Using  Image  and  Curve  Registration  for 
Measuring  the  Goodness  of  Fit  of  Spatial  and  Temporal  Predictions.  Journal  of  Biometrics,  60, 
954-964,  Dec  2004. 

•  Sandgathe,  S.A.:  An  Automated  Mesoscale  Forecast  Verification  System.  Presentation  at  20th 
Symposium  on  Weather  Analysis  and  Forecasting,  Annual  Meeting  of  the  American 
Meteorological  Society,  14  January  2004. 

•  Sandgathe,  S.A.:  An  Automated  Forecast  Verification  System.  Presentation  at  the  Northwest 
Weather  Workshop,  7  March  2003. 

•  Sandgathe,  S.A.:  An  Automated  Forecast  Verification  System  for  Mesoscale  Ensembles.  FAA 
Verification  Workshop,  30  July  2002. 

•  Sen,  Z.,  1997:  Objective  Analysis  by  Cumulative  Semivariogram  Technique  and  its 
Application  in  Turkey.  J.  Appl.  Meteo,  36,  1712-1724. 

•  Van  Galen,  J.,  1970:  A  new  method  for  verifying  deterministic  predictions  of  meteorological 
scalar  fields.  Tellus,  XXII,  32-42. 

PUBLICATIONS 

•  Marzban,  C.,  and  S.  Sandgathe,  2006a:  Cluster  analysis  for  verification  of  precipitation  fields. 
Wea.  Forecasting,  21,  5,  824-838. 

•  Marzban,  C.,  and  S.  Sandgathe,  2006b:  Cluster  analysis  for  object-oriented  verification  of 
fields:  a  variation.  Wea.  Forecasting,  (submitted) 

•  Reilly,  C.,  P.  Price,  A.  Gelman  and  S.  A.  Sandgathe,  2004:  Using  Image  and  Curve 
Registration  for  Measuring  the  Goodness  of  Fit  of  Spatial  and  Temporal  Predictions.  Journal  of 
Biometrics,  60,  954-964. 

•  Marzban,  C.,  S.  Sandgathe,  E.  Kalnay,  2005:  MOS,  Perfect  Prog,  and  Reanalysis  Data. 
Accepted  Monthly  Weather  Review. 


